Method and system for data indexing and reporting

ABSTRACT

The present teaching relates to system, method, and programming for indexing user records. A plurality of user records are obtained, wherein each user record is associated with a user identifier and one or more data-segments. For each data-segment, one or more user identifiers associated with the data-segment are determined, a relative distance of each user identifier with respect to a previous user identifier associated with the data-segment is computed, and a variable number of bits to be used to store the relative distance of each user identifier in a bit stream are determined based on a criterion associated with the relative distance. Offset values are generated for the one or more data-segments, wherein an offset value of a data-segment indicates an index of the bit stream where information of a first user identifier associated with the data-segment is stored.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 16/235,295, filed on Dec. 28, 2018, the contents of which arehereby incorporated by reference in its entirety.

BACKGROUND 1. Technical Field

The present teaching generally relates to data processing. Morespecifically, the present teaching relates to techniques of processingand indexing Big Data.

2. Technical Background

In the age of the Internet, amount of data available becomes explosive.Great effort has been made to analyze the vast amount of data to makesome sense out of it in order to improve the efficiency associated withdata access. With considerable advances in computing and storagetechnologies, advancements in cloud computing, development of ubiquitousconnectivity, and the Internet of Things, there has been an explosion inthe size and variety of collected data. Nevertheless, one can bedata-rich and knowledge-poor, and this is where data analytics and thedevelopment and application of machine learning models becomes anecessity for gaining insights of complex processes to support decisionmaking and enhance strategic planning in different areas of economy,finance, industry, healthcare, etc.

Recently, there has also been an influx of polymorphic, unstructured andmultimodal data—social media, images, audio, video, etc., which iscomplicating further the data processing and knowledge extractionprocess. In scenarios where generating real-time reports is required,the existing data analytics systems cannot process such data as theyheavily rely on pre-aggregated data rather than raw data. Specifically,traditional analytics systems are designed to work with data that hasbeen pre-processed based on a set of predetermined rules. As such, thetraditional analytics systems are adapted to provide responses toqueries of only a certain type. Moreover, the traditional analyticssystems are not designed to handle ad-hoc complex targeting rules andgenerate real-time reports based on high-dimensional data in a fast andefficient manner.

Thus, there is a need for methods and systems that address the abovestated limitations.

SUMMARY

The teachings disclosed herein relate to methods, systems, andprogramming for processing and indexing Big data.

One aspect of the present disclosure provides for a method implementedon a machine having at least one processor, storage, and a communicationplatform capable of connecting to a network for indexing user records.The method includes the steps of obtaining a plurality of user records,wherein each user record is associated with a user identifier and one ormore data-segments. For each data-segment, the method includesdetermining one or more user identifiers associated with thedata-segment, computing a relative distance of each user identifier withrespect to a previous user identifier associated with the data-segment,and determining, based on a criterion associated with the relativedistance, a variable number of bits to be used to store the relativedistance of each user identifier in a bit stream. The method includesgenerating offset values for the one or more data-segments, wherein anoffset value of a data-segment indicates an index of the bit streamwhere information of a first user identifier associated with thedata-segment is stored.

By one aspect of the present disclosure, there is provided a system forindexing user records. The system includes a data receiving unitconfigured for obtaining a plurality of user records, wherein each userrecord is associated with a user identifier and one or moredata-segments, a segment population unit configured for determining, foreach data-segment, one or more user identifiers associated with thedata-segment, a relative distance computing unit configured forcomputing, for each data-segment, a relative distance of each useridentifier with respect to a previous user identifier associated withthe data-segment, a relative distance encoder configured fordetermining, for each data-segment, a variable number of bits to be usedto store the relative distance of each user identifier in a bit streambased on a criterion associated with the relative distance, and a datastream generator configured for generating offset values for the one ormore data-segments, wherein an offset value of a data-segment indicatesan index of the bit stream where information of a first user identifierassociated with the data-segment is stored.

Other concepts relate to software for implementing the present teaching.A software product, in accord with this concept, includes at least onemachine-readable non-transitory medium and information carried by themedium. The information carried by the medium may be executable programcode data, parameters in association with the executable program code,and/or information related to a user, a request, content, or otheradditional information.

In one example, there is provided, a non-transitory computer readablemedium including computer executable instructions, wherein theinstructions, when executed by a computer, cause the computer to performa method for indexing user records. The method includes the steps ofobtaining a plurality of user records, wherein each user record isassociated with a user identifier and one or more data-segments. Foreach data-segment, the method includes determining one or more useridentifiers associated with the data-segment, computing a relativedistance of each user identifier with respect to a previous useridentifier associated with the data-segment, and determining, based on acriterion associated with the relative distance, a variable number ofbits to be used to store the relative distance of each user identifierin a bit stream. The method includes generating offset values for theone or more data-segments, wherein an offset value of a data-segmentindicates an index of the bit stream where information of a first useridentifier associated with the data-segment is stored.

Additional advantages and novel features will be set forth in part inthe description which follows, and in part will become apparent to thoseskilled in the art upon examination of the following and theaccompanying drawings or may be learned by production or operation ofthe examples. The advantages of the present teachings may be realizedand attained by practice or use of various aspects of the methodologies,instrumentalities and combinations set forth in the detailed examplesdiscussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The methods, systems and/or programming described herein are furtherdescribed in terms of exemplary embodiments. These exemplary embodimentsare described in detail with reference to the drawings. Theseembodiments are non-limiting exemplary embodiments, in which likereference numerals represent similar structures throughout the severalviews of the drawings, and wherein:

FIGS. 1A-1C depict different operational configurations of a dataindexing engine in a network setting, according to different embodimentsof the present teaching;

FIG. 2 depicts an exemplary high-level system diagram of a data indexingengine, according to an embodiment of the present teaching;

FIG. 3 is a flowchart of an exemplary process of a data indexing engine,according to an embodiment of the present teaching;

FIG. 4 depicts an exemplary high-level system diagram of an indexingcluster, according to an embodiment of the present teaching;

FIG. 5 is a flowchart of an exemplary process of an indexing cluster,according to an embodiment of the present teaching;

FIG. 6 depicts an exemplary high-level system diagram of an indexprocessing unit, according to an embodiment of the present teaching;

FIG. 7 is a flowchart of an exemplary process of an index processingunit, according to an embodiment of the present teaching;

FIG. 8 is a schematic depicting data-segments associated with userrecords;

FIGS. 9A-9F illustrate schematics of indexing and storing user records,according to embodiments of the present teaching;

FIG. 10 depicts an architecture of a mobile device which can be used toimplement a specialized system incorporating the present teaching; and

FIG. 11 depicts the architecture of a computer which can be used toimplement a specialized system incorporating the present teaching.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, well known methods, procedures, components,and/or circuitry have been described at a relatively high-level, withoutdetail, in order to avoid unnecessarily obscuring aspects of the presentteachings.

Subject matter will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific example embodiments.Subject matter may, however, be embodied in a variety of different formsand, therefore, covered or claimed subject matter is intended to beconstrued as not being limited to any example embodiments set forthherein. Example embodiments are provided merely to be illustrative.Likewise, a reasonably broad scope for claimed or covered subject matteris intended. Among other things, for example, subject matter may beembodied as methods, devices, components, or systems. Accordingly,embodiments may, for example, take the form of hardware, software,firmware or any combination thereof (other than software per se). Thefollowing detailed description is, therefore, not intended to be takenin a limiting sense.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in one embodiment” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterinclude combinations of example embodiments in whole or in part.

In general, terminology may be understood at least in part from usage incontext. For example, terms, such as “and”, “or”, or “and/or,” as usedherein may include a variety of meanings that may depend at least inpart upon the context in which such terms are used. Typically, “or” ifused to associate a list, such as A, B or C, is intended to mean A, B,and C, here used in the inclusive sense, as well as A, B or C, here usedin the exclusive sense. In addition, the term “one or more” as usedherein, depending at least in part upon context, may be used to describeany feature, structure, or characteristic in a singular sense or may beused to describe combinations of features, structures or characteristicsin a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again,may be understood to convey a singular usage or to convey a pluralusage, depending at least in part upon context. In addition, the term“based on” may be understood as not necessarily intended to convey anexclusive set of factors and may, instead, allow for existence ofadditional factors not necessarily expressly described, again, dependingat least in part on context.

FIGS. 1A-1C depict different operational configurations of a dataindexing engine in a network setting, according to different embodimentsof the present teaching. In FIG. 1A, an exemplary system configuration100 includes users 110, a network 120, an exemplary publisher 130,content sources 160 including content source 1 160-a, content source 2160-b, . . . , content source n 160-c, an advertisement server 140, anda data indexing engine 170. In this illustrated embodiment, the dataindexing engine 170 is directly connected to the network 120 andoperates as an independent service engine that processes and indexesdata.

By one embodiment of the present teaching, data processed by the dataindexing engine 170 may relate to user records that each contain severalvariable length arrays. Each array is referred to herein as adata-segment and may correspond to data related to mail, purchases madeby the user, search events related to the user, campaign events,advertisement clicking activities of the user, etc. The data indexingengine 170 may receive a query related to the user records, forinstance, from the publisher 130. As discussed herein, the data indexingengine 170 processes the user records and indexes (and stores) data in aspecific manner that enables the data indexing engine 170 to generate aresponse to the query in a fast and efficient manner. For example, thedata indexing engine 170 may generate a plurality of reports and providethe reports as response(s) to the query to the publisher 130 via thenetwork 120.

In the embodiment illustrated in FIG. 1A, as the data indexing engine170 is a stand-alone service, it may provide its services to a pluralityof publishers 130 and a plurality of advertisement servers 140 (notshown plurality of each). It must be appreciated that in someapplications, the data indexing engine 170 may also be used to issueresponses to queries issued by other parties e.g., advertisers,third-party service providers, etc.

In FIG. 1B, an alternative configuration 200 is provided, in which thedata indexing engine 170 is connected to a publisher 130 as its backendservice engine. That is, in this embodiment, the data indexing engine170 is a special module in the backend of the publisher 130. When thereare multiple publishers (not shown), each may have its own backendmodule for processing and indexing data and providing responses toqueries in real time.

In FIG. 1C, yet another alternative configuration 300 is provided, inwhich the data indexing engine 170 is connected to an advertisementserver 140 as a backend service engine. That is, in this embodiment, thedata indexing engine 170 is a special module in the backend of theadvertisement server 140. When there are multiple advertisement servers,each may have its own backend module for processing and indexing dataand providing responses to queries in real time.

An application of the data indexing engine 170 as depicted in theembodiments illustrated in FIG. 1B and 1C may be related to processinguser records that include advertisement activities of the users. Forinstance, Ads stored in an advertisement database 150 may be provided(via the Ad server 140) to users on a webpage associated the publisher130. The publisher 130 and/or advertisers (or third-party vendors) maybe interested in qualifying users based on certain advertisement eventsor other criteria. In such cases, the publisher 130 and/or advertisersmay issue a query to the data indexing engine 170 to obtain responses totheir specific queries. Accordingly, the data indexing engine 170processes the user records and indexes data in a specific manner thatenables it to generate the response to the queries in a fast andefficient manner.

In FIGS. 1A-1C, the network 120 may be a single network or a combinationof different networks. For example, a network may be a local areanetwork (LAN), a wide area network (WAN), a public network, a privatenetwork, a proprietary network, a Public Telephone Switched Network(PSTN), the Internet, a wireless network, a cellular network, aBluetooth network, a virtual network, or any combination thereof. Thenetwork 120 may also include various network access points, e.g., wiredor wireless access points such as base stations 120-a, 120-b, orInternet exchange points (not shown) through which a data source mayconnect to the network 120 in order to transmit/receive information viathe network.

In some embodiments, the network 120 may be an online advertisingnetwork or an ad network, which connects the data indexing engine 170to/from the publisher 130 or websites/mobile applications hosted thereonthat desire to receive or display advertisements. Functions of an adnetwork may include an aggregation of ad-space supply from the publisher130, ad supply from the advertisement server 140, etc. An ad network maybe any type of advertising network environments such as a television adnetwork, a print ad network, an online (Internet) ad network, or amobile ad network.

The publisher 130 can be a content provider, a search engine, a contentportal, or any other sources from which content can be published. Thepublisher 130 may correspond to an entity, whether an individual, afirm, or an organization, publishing or supplying content, including ablogger, television station, a newspaper issuer, a web page host, acontent portal, an online service provider, or a game server. Forexample, in connection to an online or mobile ad network, publisher 130may also be an organization such as USPTO.gov and CNN.com, or a contentportal such as YouTube and Yahoo.com, or a content-soliciting/feedingsource such as Twitter, Facebook, or blogs. In one example, content sentto a user may be generated or formatted by the publisher 130 based ondata provided by or retrieved from the content sources 160.

The content sources 160 may correspond to content/app providers, whichmay include, but not limited to, to an individual, a business entity, ora content collection agency such as Twitter, Facebook, or blogs, thatgather different types of content, online or offline, such as news,papers, blogs, social media communications, magazines, whether textual,audio visual such as images or video content. The publisher may also bea content portal presenting content originated by a different entity(either an original content generator or a content distributor).Examples of a content portal include, e.g., Yahoo! Finance, Yahoo!Sports, AOL, and ESPN. The content from content sources 160 includemulti-media content or text or any other form of content includingwebsite content, social media content from, e.g., Facebook, Twitter,Reddit, etc., or any other content generators. The gathered content maybe licensed content from providers such as AP and Reuters. It may alsobe content crawled and indexed from various sources on the Internet.Content sources 160 provide a vast range of content that are searchableor obtainable by the publisher 130.

Users 110 may be of different types such as ones connected to thenetwork via wired or wireless connections via a device such as a desktop110-d, a laptop 110-c, a handheld device 110-a, a built-in deviceembedded in a vehicle such as a motor vehicle 110-b, or wearable devices(e.g., glasses, wrist watch, etc.). In one embodiment, users 110 may beconnected to the network 120 to access and interact with online content(provided by the publisher 130), via wired or wireless means, throughrelated operating systems and/or interfaces implemented within therelevant user interfaces.

By one embodiment of the present teaching, in operation, a request foran advertisement from the publisher 130 is received by the advertisementserver 140, which may be centralized or distributed. The advertisementserver 140 may archive data related to a plurality of advertisements inan advertisement database 150, which may or may not reside in the cloud.The advertisement server 140 operates to distribute advertisements toappropriate ad placement opportunities on different platforms. Theadvertisements accessible by the advertisement server 140 may includesome textual information, e.g., a description of what the advertisementis about as well as additional information such as target audience aswell as certain distribution criteria related to, e.g., geographicalcoverage or timing related requirements. Target audience may bespecified in terms of, e.g., demographics of the target audience, thedistribution criteria may specify geographical locations of the targetaudience, and/or time frame(s) the advertisement is to be distributed tothe target audience.

When a request is received from the publisher 130 for an advertisement,either the publisher 130 or the advertisement server 140 may invoke thedata indexing engine 170 to process data streams (e.g., a data streamincluding information related to user interaction with online content)and provide responses to queries. In what follows, there is provided adetailed description of the data indexing engine 170 and techniquesutilized therein to index data in an efficient manner such thatresponses to queries can be generated in real-time.

FIG. 2 depicts an exemplary high-level system diagram of a data indexingengine 170, according to an embodiment of the present teaching. The dataindexing engine 170 includes an application interface (API) 210 viawhich a user 205 communicates with the data indexing engine 170. Thedata indexing engine 170 also includes a query allocation unit 220, aquery allocation controller 230, a plurality of indexing clusters 250including indexing cluster 1 250-1, . . . , indexing cluster K 250-2, atriggering unit 260, a plurality of data ingestion controllers 270including data ingestion controller 1 270-1, . . . data ingestioncontroller M 270-2, a scheduling unit 280, and data networks 290 whichinclude for instance a grid network 290-1, a Hadoop file system 290-2,etc.

According to an embodiment of the present teaching, data related to auser is referred to herein as a user record. Each user record isassociated with a unique user-identifier (ID). The contents of a userrecord may include single attribute data (e.g., age of the user, genderof the user, etc.), as well as several kinds of arrays (each referred toherein as a segment array or data-segment) that include informationrelated to, for instance, user activity such as purchases, clickingactivity, impressions, search activity, etc. Such user records arestored in the data networks 290.

The data ingestion controllers 270 retrieve data stored in the datanetworks 290 and transmit the retrieved data to the indexing clusters250 for further processing. The data ingestion controllers 270 areprogrammed to obtain snapshots of the data stored in the data networks290. For example, the data ingestion controllers 270-1, 270-2 etc., maybe configured to obtain a snapshot of the data stored in the gridnetwork 290-1, Hadoop file system 290-2 etc., after expiration of apredetermined amount of time (e.g., 30 minutes, 2 hours, 4 hours, etc.).Alternatively, or in addition to, the data ingestion controllers 270 maybe activated by the triggering unit 260 to perform the data retrievalprocess from the data networks 290. Specifically, the triggering unit260 may serve as an alternate means (e.g., external third-partytriggers) that trigger the data ingestion controllers 270 to retrievethe data from the data networks 290.

Upon retrieving the data from the data networks 290, the scheduling unit280 schedules the transmission of the retrieved data from the dataingestion controllers 270 to the plurality of indexing clusters 250. Byone embodiment of the present teaching, the scheduling unit 280 mayschedule data transmission to the indexing clusters 250 based on ascheduling criterion such as at predetermined time-intervals, ascheduling priority based on the type of data, etc. Each indexingcluster 250-1, 250-2 etc., is configured to obtain the data from thedata ingestion controllers 270 and further process (and store) the datain a manner such that responses to real-time queries (related to thedata) can be generated in a fast and efficient manner e.g., responses tothe real-time queries can be generated in order of milli-seconds.

By one embodiment of the present teaching, each indexing cluster 250-1,250-2 etc., is configured to process and store data in a column-basedformat. Each column may be stored in a separate binary file. Thus, whileprocessing queries, only relevant data (i.e., binary files) may beloaded and processed to generate the response to the query in a fastmanner. It must be appreciated that the column-based storage format asimplemented by the indexing clusters 250 provides high performanceimprovements as compared to traditional row-based storage systems.Details regarding the indexing clusters 250 is provided next withreference to FIGS. 4 and 6 .

As stated previously, the user 205 communicates with the data indexingengine 170 via the API 210. A query submitted by the user 205 isreceived by the query allocation unit 220. The query allocationcontroller 230 controls the query allocation unit 220 and directs thereceived query to at least one indexing cluster 250. Specifically, thequery allocation controller 230 utilizes allocation criteria 240 todirect the query to at least one of the indexing clusters 250. Theallocation criteria may correspond to for instance, a current processingworkload of the indexing cluster and the like. The indexing cluster thatreceives the query utilizes the processed data (i.e., data stored in thecolumn-based format) to generate a response to the query. The generatedresponse is provided by the indexing cluster to the user 205 via the API210. It must be appreciated that the teachings of the present disclosureare in no way limited to the specific configuration of the data indexingengine 170 as depicted in FIG. 2 . For instance, although a single API210 is depicted in FIG. 2 , the data indexing engine 170 may includemultiple APIs (e.g., application specific interfaces) via which usersmay communicate with the data indexing engine 170.

FIG. 3 is a flowchart of an exemplary process of a data indexing engine,according to an embodiment of the present teaching. The method commencesin step 310 wherein a query is received from a user. In step 320, thereceived query is assigned to one or more indexing clusters based on acriterion.

In step 330, raw data stored in data networks is obtained via the dataingestion controllers based on triggering conditions. In step 340, thedata indexing engine may pre-process the obtained raw data. For example,the data indexing engine may pre-process the obtained raw data totransform the raw data in a format that is suitable for indexingpurposes.

The method then proceeds to step 350, wherein the pre-processed data isscheduled for transmission to the indexing clusters. In step 360, theindexing clusters of the data indexing engine process the data i.e.,index and store the data in a particular format such that response(s) tothe query issued by the user can be generated in an efficient manner. Instep 370, the generated response is provided (e.g., via the API) to theuser.

FIG. 4 depicts an exemplary high-level system diagram of an indexingcluster 250-1, according to an embodiment of the present teaching. Itmust be appreciated that each indexing cluster of the plurality ofindexing clusters 250 has a configuration similar to that as depicted inFIG. 4 . The indexing cluster 250-1 includes a data portioning unit 410,a data allocation unit 420, a plurality of index processing units 430-1,430-2 . . . , 430-3, a merging unit 440, and a response generating unit450.

The data portioning unit 410 receives data from the data ingestioncontrollers 270, which in turn retrieve data from the data networks 290(as shown in FIG. 2 ). The data partitioning unit 410 utilizespartitioning rules 405 to partition the received data. For example, thedata partitioning unit 410 may uniformly partition the received datasuch that the data allocation unit 420 may allocate equal (or nearlyequal) amounts of data to each of the index processing units 430.Alternatively, the data partitioning unit 410 may partition the receiveddata based on a current processing workload of each index processingunit 430-1, 430-2, . . . 430-3. Specifically, a workload database 403maintains information related to the processing workload of each indexprocessing unit 430-1, 430-2, . . . 430-3. The data partitioning unit410 may partition (and further allocate via the data allocation unit420) the received data in a manner such that each index processing unit430-1, 430-2, . . . 430-3 has a similar workload.

Each of the index processing units 430-1, 430-2, . . . 430-3 processesthe received data and indexes the data in a specific manner such thatresponses to queries may be generated in a fast and efficient manner.Details regarding the index processing units 430 are described next withreference to FIG. 6 . The responses (e.g., results obtained via theprocessing of the individual indexed data of the index processing units430) generated from each of the index processing units 430-1, 430-2, . .. 430-3 are forwarded to the merging unit 440 that is configured tointegrate the individual responses. Further, the response generatingunit receives the integrated results from the merging unit 440 andutilizes templates 450 to generate for instance, real-time reportsrelated to the query(s). For example, by one embodiment of the presentteaching, the response generating unit 450 may utilize templates such asfrequency distribution reports, event related reports, etc. Thegenerated report(s) may be transmitted to the API 210 as shown in FIG. 2to be presented to the user.

FIG. 5 depicts a flowchart of an exemplary process of an indexingcluster, according to an embodiment of the present teaching. The processcommences in step 510 wherein the indexing cluster receives data fromthe data ingestion controllers. In step 520, the received data ispartitioned based on partitioning criteria. The process then moves tostep 530, wherein the partitioned data is allocated to the indexprocessing units.

In step 540, the index processing units further process the allocateddata and index the data in a specific manner that enhances dataprocessing performance of the indexing engine. For instance, by oneembodiment of the present teaching, the index processing units compressand store data in a particular format that balances storage usage,data-compression ratio, and reporting performance. Details regardingthese features are described next.

The process then proceeds to step 550 wherein each index processing unitprocesses a query based on its respective indexed data. In step 560, theresults of processing the query are merged. In step 570, the indexingcluster generates a report(s) (via utilizing templates) that are furthertransmitted to an API to eventually be presented to a user.

Turning now to FIG. 6 , there is depicted an exemplary high-level systemdiagram of an index processing unit 430-1, according to an embodiment ofthe present teaching. It must be appreciated that each indexingprocessing unit included in the indexing cluster (as shown in FIG. 4 )has a configuration similar to that as depicted in FIG. 6 . By oneembodiment, the index processing unit 430-1 is configured to process andindex the data in a unique manner. Specifically, the index processingunit 430-1 stores indexed data in a column format. As stated previously,each column may be stored in a separate binary file. Thus, whileprocessing queries, only relevant data (i.e., binary files) may beloaded and processed to generate a response to the query in a fastfashion. The index processing unit 430-1 includes a data preprocessingunit 605, a segment population unit 610, a relative distance computingunit 620, a comparator 630, a relative distance encoder 625, a datastream generator 645, and a query processing unit 655.

The data processing unit 605 receives data and pre-processes thereceived data. By one embodiment, the received data may be user recordsas shown in FIG. 8 . Each user record is associated with a useridentifier (ID) and one or more data-segments 810-a, 810-b . . . ,810-c. Each data-segment is associated with a segment ID 812. Eachdata-segment may correspond to data related to mail, purchases made bythe user, search events related to the user, campaign events,advertisement clicking activities of the user, etc.

The user records may be pre-processed by the data pre-processing unit605, for instance, to ensure that information related to thedata-segments of the user records e.g., segment IDs and the like, arevalid. The pre-processed data is forwarded to the segment populationunit 610. The segment population unit 610 is configured to generatesegment data 615. By one embodiment of the present teaching, the segmentpopulation unit 610 receives as input, pre-processed user records asshown in FIG. 9A. Specifically, FIG. 9A depicts users records and theirassociated data-segments maintained in a tabular format. The segmentpopulation unit 610 generates segment data 615 by performing a transposeoperation of the data depicted in FIG. 9A. In other words, the segmentpopulation unit 610 identifies for each data-segment, a first user IDand additional user IDs associated with the data-segment. The segmentdata 615 may be maintained in a tabular format 915 as shown in FIG. 9B.

The segment data 615 is forwarded to the relative distance computingunit 620, which is configured to compute for each data-segment, arelative distance of a user ID with respect to a previous user ID thatis associated with the data-segment. For instance, referring to FIG. 9Bit can be seen that for data-segment having segment ID 10, the usersassociated with the data segment have user IDs of: 0 (first user IDassociated with the data-segment 10), and 1, 100, 101, and 200(additional user IDs associated with data-segment 10). By one embodimentof the present teaching and as depicted in FIG. 9C, the relativedistance computing unit 620 computes the relative distance of theadditional user IDs associated with each data-segment. As shown in FIG.9C, with regard to the data-segment having segment ID 10, the relativedistance computing unit 620 computes the relative distances of +1, +99,+1, and +99 for the additional user IDs 1, 100, 101, and 200 that areassociated with the data-segment 10. Furthermore, the relative distancecomputing unit 620 may store the computed relative distances in atabular format as shown in FIG. 9C.

By one embodiment of the present teaching, the index processing unit430-1 encodes each computed relative distance based on a criterionassociated with a magnitude of the relative distance. Specifically,based on the relative distance being in a first range (i.e., 0-127), theindex processing unit encodes the relative distance as the magnitude ofrelative distance plus a predetermined first number (e.g. 128) into asingle byte. For example, as shown in FIG. 9D, for a relative distance(between user IDs) of magnitude 17, the index processing unit encodesthe relative distance as: 17+128=145 into binary format represented as‘10010001’. Further, based on the relative distance being in a secondrange (i.e., 128-16383), the index processing unit encodes the relativedistance as the magnitude of relative distance plus a predeterminedsecond number (e.g. 32768) into two bytes. For example, as shown in FIG.9D, for a relative distance (between user IDs) of magnitude 1025, theindex processing unit encodes the relative distance as: 1025+32768=33793into binary format represented as ‘01000100 00000001’. Similarly, basedon the relative distance being in a third range (e.g., greater than16384), the index processing unit encodes the value of the relativedistance into three bytes.

Referring back to FIG. 6 , each computed relative distance is forwardedto the comparator 630. The comparator 630 utilizes thresholds 635 todetermine a range in which the computed relative distance lies in. Basedon the range in which the relative distance lies, an encoding parameterof the relative distance encoder 625 is selected to encode the relativedistances as described above. In this manner, the relative distanceencoder 625 encodes the relative distances (of the user IDs) for eachdata-segment to generate encoded segment data 640, which can bemaintained in a tabular format 930 as shown in FIG. 9E. Moreover, theuser ID of the first user associated with each data-segment is encodedto a fix number of bits (e.g., three bytes). That is the user ID isexpressed in a binary format having a fixed length e.g. 24 bits.

Upon completing the encoding of the relative distances of user IDsassociated with the data-segments, the data stream generator 645retrieves the encoded data 640 to generate a byte stream and an offsetdata stream 650. Specifically, by one embodiment of the presentteaching, the data stream generator 645 compiles a single byte stream ofuser IDs relative distances associated with the data-segments. Forexample, referring to FIG. 9F, table 950 depicts the byte stream whichincludes encoded relative distances of user IDs with respect to threedata-segments having respective segment IDs denoted as 10, 2000, and30000. It must be appreciated that the byte stream 950 containscompressed data (i.e., a byte array) containing a variable number ofbytes associated with each relative distance of the user ID. In otherwords, as described before, each relative distance of the user ID may beencoded as one of a single byte, two bytes, and three bytes based on amagnitude of the relative distance.

Moreover, the data stream generator 645 also generates an offset stream,which includes information pertaining to offset of user-recordsassociated with the data-segments. Such an offset stream may correspondto header information associated with the byte stream. For example,referring to FIG. 9F, table 940 depicts the offset stream informationincluding an index (i.e., an identifier corresponding to the startinglocation of the data-segment within the byte stream) for eachdata-segment. As shown in FIG. 9F, data-segment having a segment ID of10 has an offset of 0, whereas data-segment having a segment ID of 2000has an offset of 7. The respective offset values correspond to the index(of the byte stream) where information of a first user identifierassociated with the data-segment is stored. Further, the queryprocessing unit 655 receives a query issued by a user. The queryprocessing unit 655 can utilize the byte and offset data streams 650 togenerate a response to the query which is eventually provided to theuser.

FIG. 7 depicts a flowchart of an exemplary process of an indexprocessing unit, according to an embodiment of the present teaching. Theprocess commences in step 710, wherein user data (i.e., user records)are obtained. In step 720, the information related to data-segments isgenerated. Specifically, for each data-segment, the user IDs associatedwith the data-segment are identified.

In step 730, for each data-segment, a relative distance of each user IDwith respect to a previous user ID associated with the data-segment iscomputed. Further, the process moves to step 740, wherein the computedrelative distances are encoded based on an encoding criterion. Forexample, as stated previously, the relative distance may be encodedbased on a range in which the relative distance lies in. Further, instep 750, a byte stream and an offset stream are generated based on theencoded relative distances. Thereafter, in step 760, a response to aquery is generated based on the byte stream.

It must be appreciated that the data compression mechanism (i.e., theencoding technique) described herein is effective and optimized for userrecords. If a data-segment is large (i.e., the data segment isassociated with a large number of users) then the average distancebetween the user IDs is small, and thus only a single byte is requiredto store the data. For a moderate sized data-segment, the averagerelative distances may be for example 14 bits long, and thus may fitinto two bytes. Moreover, if the data-segment is very small in size(i.e., the data-segment is associated with fewer number of user IDs),then the individual positions may be stored using three bytes of data,which is still considerably smaller than storing the original IDs, whichare typically at least 64 bits long. Moreover, performance of generatinga response to a query is also enhanced due to the index processing unitas the indexing cluster can access and load only relevant user IDs (fromthe byte stream) related to any data-segment.

Additionally, in the encoding mechanism as described above, it must benoted that the first two bits of the first encoded byte associated withthe user signify a size (number of bytes) used to encode the user ID'srelative distance. Specifically, if the first bit is one then the blocksize (i.e., the number of bytes used to encode the relative distance ofthe user ID) is one byte, whereas if the second bit is one, then theblock size is two bytes, else the block size is 3 bytes.

Accordingly, the data indexing engine of the present disclosure canobtain user IDs related to a segment, as well as obtain data-segmentsassociated with a particular user in a quick and seamless manner. Thedata indexing engine of the present teaching provides for evaluatingcomplex ad-hoc targeting rules and generating real-time reports based onhigh-dimensional user data. Moreover, it must be appreciated that theteachings of the present disclosure are in no way limited to thespecific implementation as described above. For instance, according toone embodiment of the present teaching, the encoding of the relativedistances of the user IDs may be performed in manner (e.g., based on themagnitude of the relative distances) such that the relative distance ofa user ID can be encoded into a variable number of bits, as opposed tobytes.

Turning now to FIG. 10 , there is depicted an architecture of a mobiledevice 1000, which can be used to realize a specialized systemimplementing the present teaching. In this example, a user device onwhich the functionalities of the various embodiments described hereincan be implemented is a mobile device 1000, including, but not limitedto, a smart phone, a tablet, a music player, a handled gaming console, aglobal positioning system (GPS) receiver, and a wearable computingdevice (e.g., eyeglasses, wrist watch, etc.), or in any other formfactor.

The mobile device 1000 in this example includes one or more centralprocessing units (CPUs) 1040, one or more graphic processing units(GPUs) 1030, a display 1020, a memory 1060, a communication platform1010, such as a wireless communication module, storage 1090, and one ormore input/output (I/O) devices 1050. Any other suitable component,including but not limited to a system bus or a controller (not shown),may also be included in the mobile device 1000. As shown in FIG. 10 , amobile operating system 1070, e.g., iOS, Android, Windows Phone, etc.,and one or more applications 1080 may be loaded into the memory 1060from the storage 1090 in order to be executed by the CPU 1040. Theapplications 1080 may include a browser or any other suitable mobileapps for performing the various functionalities on the mobile device1000. User interactions with the content displayed on the display panel1020 may be achieved via the I/O devices 1050.

To implement various modules, units, and their functionalities describedin the present disclosure, computer hardware platforms may be used asthe hardware platform(s) for one or more of the elements describedherein. The hardware elements, operating systems and programminglanguages of such computers are conventional in nature, and it ispresumed that those skilled in the art are adequately familiar therewithto adapt those technologies. A computer with user interface elements maybe used to implement a personal computer (PC) or other type of workstation or terminal device, although a computer may also act as a serverif appropriately programmed. It is believed that those skilled in theart are familiar with the structure, programming, and general operationof such computer equipment and as a result the drawings should beself-explanatory.

FIG. 11 is an illustrative diagram of an exemplary computer systemarchitecture, in accordance with various embodiments of the presentteaching. Such a specialized system incorporating the present teachinghas a functional block diagram illustration of a hardware platform whichincludes user interface elements. Computer 1100 may be a general-purposecomputer or a special purpose computer. Both can be used to implement aspecialized system for the present teaching. Computer 1100 may be usedto implement any component(s) described herein. For example, the presentteaching may be implemented on a computer such as computer 1100 via itshardware, software program, firmware, or a combination thereof. Althoughonly one such computer is shown, for convenience, the computer functionsrelating to the present teaching as described herein may be implementedin a distributed fashion on a number of similar platforms, to distributethe processing load.

Computer 1100, for example, may include communication ports 1150connected to and from a network connected thereto to facilitate datacommunications. Computer 1100 also includes a central processing unit(CPU) 1120, in the form of one or more processors, for executing programinstructions. The exemplary computer platform may also include aninternal communication bus 1110, program storage and data storage ofdifferent forms (e.g., disk 1170, read only memory (ROM) 1130, orrandom-access memory (RAM) 1140), for various data files to be processedand/or communicated by computer 1100, as well as possibly programinstructions to be executed by CPU 1120. Computer 1100 may also includean I/O component 1160 supporting input/output flows between the computerand other components therein such as user interface elements 1180.Computer 1100 may also receive programming and data via networkcommunications.

Hence, aspects of the present teaching(s) as outlined above, may beembodied in programming. Program aspects of the technology may bethought of as “products” or “articles of manufacture” typically in theform of executable code and/or associated data that is carried on orembodied in a type of machine readable medium. Tangible non-transitory“storage” type media include any or all of the memory or other storagefor the computers, processors or the like, or associated modulesthereof, such as various semiconductor memories, tape drives, diskdrives and the like, which may provide storage at any time for thesoftware programming.

All or portions of the software may at times be communicated through anetwork such as the Internet or various other telecommunicationnetworks. Such communications, for example, may enable loading of thesoftware from one computer or processor into another, for example, froma management server or host computer of the data indexing engine intothe hardware platform(s) of a computing environment or other systemimplementing a computing environment or similar functionalities inconnection with indexing content. Thus, another type of media that maybear the software elements includes optical, electrical andelectromagnetic waves, such as used across physical interfaces betweenlocal devices, through wired and optical landline networks and overvarious air-links. The physical elements that carry such waves, such aswired or wireless links, optical links or the like, also may beconsidered as media bearing the software. As used herein, unlessrestricted to tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

Hence, a machine-readable medium may take many forms, including but notlimited to, a tangible storage medium, a carrier wave medium or physicaltransmission medium. Non-volatile storage media include, for example,optical or magnetic disks, such as any of the storage devices in anycomputer(s) or the like, which may be used to implement the system orany of its components as shown in the drawings. Volatile storage mediainclude dynamic memory, such as a main memory of such a computerplatform. Tangible transmission media include coaxial cables; copperwire and fiber optics, including the wires that form a bus within acomputer system. Carrier-wave transmission media may take the form ofelectric or electromagnetic signals, or acoustic or light waves such asthose generated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer may read programming code and/ordata. Many of these forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to a physicalprocessor for execution.

Those skilled in the art will recognize that the present teachings areamenable to a variety of modifications and/or enhancements. For example,although the implementation of various components described above may beembodied in a hardware device, it may also be implemented as a softwareonly solution—e.g., an installation on an existing server. In addition,the forecasting engine, as disclosed herein, may be implemented as afirmware, firmware/software combination, firmware/hardware combination,or a hardware/firmware/software combination.

While the foregoing has described what are considered to constitute thepresent teachings and/or other examples, it is understood that variousmodifications may be made thereto and that the subject matter disclosedherein may be implemented in various forms and examples, and that theteachings may be applied in numerous applications, only some of whichhave been described herein. It is intended by the following claims toclaim any and all applications, modifications and variations that fallwithin the true scope of the present teachings.

We claim:
 1. A method for responding to a query, comprising: obtaining aquery from a user; assigning the query to an indexing cluster of aplurality of indexing clusters based on a processing workload of theindexing cluster; obtaining data from one or more data networks based ona triggering condition or a predetermined schedule; transforming thereceived data into a column-based format; transmitting the transformeddata to the indexing cluster; and generating, by the indexing cluster, aresponse to the query based on the transformed data.
 2. The method ofclaim 1, further comprising: storing each column of the data in thecolumn-based format in a binary file.
 3. The method of claim 2, furthercomprising: processing only data stored in the binary files to generatethe response.
 4. The method of claim 1, wherein the data in thecolumn-based format comprises user records, and wherein each user recordis associated with one of a plurality of user identifiers and one ormore of a plurality of data-segments.
 5. The method of claim 4, furthercomprising: generating an offset stream utilized to generate theresponse, wherein the offset stream includes offset values for theplurality of data-segments, wherein an offset value of a correspondingdata-segment indicates an index of a bit stream where information of afirst user identifier of the plurality of user identifiers associatedwith the corresponding data-segment is stored.
 6. The method of claim 1,wherein the indexing cluster includes multiple index processing units.7. The method of claim 6, wherein each of the index processing unitsprocesses a respective portion of the transformed data to generate theresponse.
 8. A non-transitory, computer-readable medium havinginformation recorded thereon for advertising, wherein the information,when read by a machine, causes the machine to perform operationscomprising: obtaining a query from a user; assigning the query to anindexing cluster of a plurality of indexing clusters based on aprocessing workload of the indexing cluster; obtaining data from one ormore data networks based on a triggering condition or a predeterminedschedule; transforming the received data into a column-based format;transmitting the transformed data to the indexing cluster; andgenerating, by the indexing cluster, a response to the query based onthe transformed data.
 9. The medium of claim 8, wherein the operationsfurther comprise: storing each column of the data in the column-basedformat in a binary file.
 10. The medium of claim 9, wherein theoperations further comprise: processing only data stored in the binaryfiles to generate the response.
 11. The medium of claim 8, wherein thedata in the column-based format comprises user records, wherein eachuser record is associated with one of a plurality of user identifiersand one or more of a plurality of data-segments.
 12. The medium of claim11, wherein the operations further comprise: generating an offset streamutilized to generate the response, wherein the offset stream includesoffset values for the plurality of data-segments, and wherein an offsetvalue of a corresponding data-segment indicates an index of a bit streamwhere information of a first user identifier of the plurality of useridentifiers associated with the corresponding data-segment is stored.13. The medium of claim 8, wherein the indexing cluster includesmultiple index processing units.
 14. The medium of claim 13, whereineach of the index processing units processes a respective portion of thetransformed data to generate the response.
 15. A system for advertising,the system comprising: memory storing computer program instructions; andone or more processors that, in response to executing the computerprogram instructions, effectuate operations comprising: obtaining aquery from a user; assigning the query to an indexing cluster of aplurality of indexing clusters based on a processing workload of theindexing cluster; obtaining data from one or more data networks based ona triggering condition or a predetermined schedule; transforming thereceived data into a column-based format; transmitting the transformeddata to the indexing cluster; and generating, by the indexing cluster, aresponse to the query based on the transformed data.
 16. The system ofclaim 15, wherein the operations further comprise: storing each columnof the data in the column-based format in a binary file.
 17. The systemof claim 16, wherein the operations further comprise: processing onlydata stored in the binary files to generate the response.
 18. The systemof claim 15, wherein the data in the column-based format comprises userrecords, wherein each user record is associated with one of a pluralityof user identifiers and one or more of a plurality of data-segments. 19.The system of claim 18, wherein the operations further comprise:generating an offset stream utilized to generate the response, whereinthe offset stream includes offset values for the plurality ofdata-segments, and wherein an offset value of a correspondingdata-segment indicates an index of a bit stream where information of afirst user identifier of the plurality of user identifiers associatedwith the corresponding data-segment is stored.
 20. The system of claim15, wherein the indexing cluster includes multiple index processingunits, wherein each of the index processing units processes a respectiveportion of the transformed data to generate the response.